On the Efficiency of Recurrent Neural Network Optimization Algorithms
نویسندگان
چکیده
This study compares the sequential and parallel efficiency of training Recurrent Neural Networks (RNNs) with Hessian-free optimization versus a gradient descent variant. Experiments are performed using the long short term memory (LSTM) architecture and the newly proposed multiplicative LSTM (mLSTM) architecture. Results demonstrate a number of insights into these architectures and optimization algorithms, including that Hessian-free optimization has the potential for large efficiency gains in a highly parallel setup.
منابع مشابه
An efficient one-layer recurrent neural network for solving a class of nonsmooth optimization problems
Constrained optimization problems have a wide range of applications in science, economics, and engineering. In this paper, a neural network model is proposed to solve a class of nonsmooth constrained optimization problems with a nonsmooth convex objective function subject to nonlinear inequality and affine equality constraints. It is a one-layer non-penalty recurrent neural network based on the...
متن کاملPerformance Analysis of a New Neural Network for Routing in Mesh Interconnection Networks
Routing is one of the basic parts of a message passing multiprocessor system. The routing procedure has a great impact on the efficiency of a system. Neural algorithms that are currently in use for computer networks require a large number of neurons. If a specific topology of a multiprocessor network is considered, the number of neurons can be reduced. In this paper a new recurrent neural ne...
متن کاملPerformance Analysis of a New Neural Network for Routing in Mesh Interconnection Networks
Routing is one of the basic parts of a message passing multiprocessor system. The routing procedure has a great impact on the efficiency of a system. Neural algorithms that are currently in use for computer networks require a large number of neurons. If a specific topology of a multiprocessor network is considered, the number of neurons can be reduced. In this paper a new recurrent neural ne...
متن کاملInvestigation of potato peel-based bio-sorbent efficiency in reactive dye removal: Artificial neural network modeling and genetic algorithms optimization
Over the last few years, a number of investigations have been conducted to explore the low cost sorbents for the decontamination of toxic materials. Undoubtedly, agricultural waste mass is presently one of the most challenging topics, which has been gaining attention during the past several decades. Wastes are very cheap and easily available material in production of sorbent. Therefore, the Rea...
متن کاملVMLP neural network design using optimization algorithms to predict spider suspend (Case Study: Watershed Dam Kardeh)
One of the most important processes of erosion and sediment transport in streams is the river most complex engineering issues.this process special effects on water quality indices, action suburbs floor and destroyed much damage to the river and also into the development plans Lack of continuity sediment sampling and measurement of many existing stations. due to the low number of hydrometric s...
متن کاملSolving Linear Semi-Infinite Programming Problems Using Recurrent Neural Networks
Linear semi-infinite programming problem is an important class of optimization problems which deals with infinite constraints. In this paper, to solve this problem, we combine a discretization method and a neural network method. By a simple discretization of the infinite constraints,we convert the linear semi-infinite programming problem into linear programming problem. Then, we use...
متن کامل